One or more aspects relate, in general, to processing within a computing environment, and in particular, to processing associated with memory accesses.
Computer processors typically operate at higher clock speeds than main memory. To compensate, modern processors employ caches, which are fast local memory banks most often organized in a hierarchy, that aim to hold the most frequently accessed items. Caches are typically faster than main memory, but also smaller. As such, caches are limited resources, and algorithms are used to govern which parts of main memory are to be held in the cache hierarchy. These algorithms are complex, but can be summarized as seeking to place in a cache the data most recently accessed in main memory on the assumption that recently accessed data will be accessed again in the near future. This is temporal locality. Cache algorithms also seek to place in a cache data that is near the data most recently accessed in main memory, on the assumption that data near the most recently accessed data will be accessed in the near future also. This is spatial locality.
Given that the cache is a limited resource and usually cannot contain all of the data a program will access, data that is newly stored in the cache will need to supplant existing data. Most cache algorithms therefore choose to supplant the least recently accessed data in order to make room for the most recently accessed data and adjacent data. Data in this context also refers to the instruction sequences that define a program and are executed when the program is run. These instructions are also captured in the cache much the same way despite being accessed implicitly by the processor, rather than explicitly by the program.
Since cache algorithms seek to take advantage of expected temporal and spatial locality, a lack of temporal and/or spatial locality results in wasting space in the cache hierarchy by storing data that will likely not be accessed again in the near future and supplanting data that may still be accessed in the near future. As such, optimizing a software program to best make use of the cache hierarchy involves attempting to access and operate on small collections of data at any given time and grouping related data closer together. This, however, is not always possible; programs may need to access large collections of data and/or data that is not grouped together, to the detriment of performance. Programs may also occasionally need to access data that is infrequently accessed due to a temporary condition that may shortly subside. Accessing this data has the secondary effect of likely supplanting frequently accessed data from the cache, such that when the temporary condition subsides and the program returns to accessing the previous data, that data will no longer be in the cache, thereby effectively doubling the detrimental performance effects of having to access main memory to retrieve the data.
A program's instructions may also be optimized for temporal and spatial locality in much the same way as data, by limiting the number of instructions executed at any given time and grouping related parts of the program closer together.